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S& Presentation Overview 


¢ Background 

¢ Simulation environment components 

¢ Key results of first three HITLs that compared different DAA maneuver 
guidance and display configurations 


— Key metrics used to inform SC-228 DAA MOPS: 


* Total response time: the time from when a DAA alert appears on the DAA display until 
the pilot uploads a final resolution maneuver 


* Proportion of losses of well clear: the proportion of encounters that were predicted to 
lose well clear that resulted in an actual loss of well clear 


¢ Implications of results on SC-228 MOPS 
© V&V HITL 
¢ Summary of Contributions to Phase 1 MOPS 


Background 


S& Background 


Primary activity for the UAS-NAS Human Systems Integration (HSI) Subproject 
in Phase 1 was support of RTCA Special Committee 228 Minimum Operational 
Performance Standards (MOPS) 


Provide data on the effect of various Detect and Avoid (DAA) display features 
with respect to pilot performance of the remain well clear function in order to 
determine the minimum requirements for DAA displays 
— What is the pilot contribution to the DAA timeline in terms of expected response 
time to detect, determine and execute a maneuver in response to an alert of a 
potential loss of well clear? 
— What configuration of DAA display elements/capabilities meets a minimum 
acceptable level of performance? 
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DAA/Remain Well Clear Timeline 


Time until CPA 
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I 
Latency ATC Interaction Pilot Aircraft Well Clear NMAC 
Time Response Time Maneuver Time Threshold 
(~35 sec) 
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TOTAL RESPONSE TIME: 
Detect Intruders 
Pilots Determine Resolution 
Negotiate Clearance with ATC and uplink 
maneuver to aircraft 


Pilot-DAA Timeline 
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S& Background 


¢ An early critical question for the Phase | MOPS for DAA systems was what, if 
any, level of DAA maneuver guidance would be required to support acceptable 
performance on maintaining well clear? 


e Phase | MOPS assumptions specify that the pilot in command will execute 
maneuvers to remain well clear 


— j.e., No automatic/autonomous DAA capability 
¢ Display types given level/type of maneuver guidance: 


— Informative: Provides essential information of a hazard that the remote pilot may 
use to develop and execute an avoidance maneuver. No maneuver guidance 
automation or decision aiding is provided to the pilot 

— Suggestive: Automation provides a range of potential resolution maneuvers to 
avoid a hazard with manual execution. An algorithm provides the pilot with 
maneuver decision aiding regarding advantageous or disadvantageous maneuvers 

— Directive: Automation provides specific recommended resolution guidance to 
avoid a hazard with manual or automated execution. An algorithm provides the 
pilot with specific maneuver guidance on when and how to perform the maneuver 


Simulation Environment 


S& Simulation Environment 


¢ Emulation of representative environment: 
UAS Ground Control Station (GCS) with DAA Display 
DAA system components: 


¢ Surveillance 

¢ Threat detection and alerting 

* Suggestive and directive guidance 
Air Traffic Control 
Simulated Manned Traffic 


¢ Integrated via NASA’s Live, Virtual, Constructive (LVC) architecture 


Simulation Environment: 
Ground Control Station (GCS) 


¢ The Vigilant Spirit Control Station (VSCS) 
developed by the Air Force Research 
Laboratory (AFRL) 
¢ Main Features: 
— Robust, flexible interface 
— Realistic control and navigation displays 
— System status and health monitoring 
— STANAG 4586 Compliant 
— Multi-UAS control with VSCS has been 
tested in simulation and flight by AFRL 
¢ Current UAS in the NAS version 
modifications/additions: 
— Single pilot — single UAS control 


— NAS-compatible database (low- and 
high- altitude charts with navigational 
aids/’ fixes”) 

— Integrated traffic display 


Simulation Environment: 
DAA System 


The Java Architecture for DAA Modeling and Extensibility (JADEM) was 
developed by the UAS in the NAS project at NASA Ames Research Center 
Main Functions: 

— Emulate surveillance parameters for various sensor types 

° e.g., ADS-B, active radar, TCAS, etc. 
Receive state information from simulated traffic and ownship 


Assign intruder alert levels based on given thresholds 


Provide maneuver guidance 
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Simulation Environment: 
Multi Aircraft Control Station (MACS) 


The Multi Aircraft Control Station 


(MACS) developed by the Airspace 
Operations Laboratory (AOL) at NASA 
Ames Research Center 

Provides emulation of ground- and 
air- side Air Traffic Control (ATC) 
operations 


Air Traffic Controller work stations 


Simulated traffic generator 
Psuedo pilot work stations 
IFR and VFR simulated traffic 


Traffic scenarios in Oakland Center 
(ZOA 40/41) airspace based on 
current day traffic patterns 


Simulation Environment: 
LVC Architecture 


SaaProc Input: 

* Traffic 

* Ownship 

SaaProc Output: 

* Intruders 

* Saa Threat Alerts and 
Resolutions 


VSCS Input: 
* Intruders 


* SAA Threat Alerts 


VSCS Output: 
* Ownship 


Stratway Bands 
Intruders 


Ownship 


SaaProc/JADEM 


Ownship 


Stratway Input: 
Ownship & * Intruders 


LVC Traffic * Ownship 


Gateway 


Stratway Output: 
* Stratway Bands Msg 


Stratway Bands 


Intruders 


Traffic: Ownship: 
* Fit State, * Fit State, 
* Fit Plan, * Fit Plan, 


ATC & PPilots Input: 


* Ownship 


ATC & Pseudo 
Pilot ATC & PPilots Output: 


System * Traffic 
(MACS) 


*Traj. Intent ° Traj. Intent 


Intruders: Fit State 
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HITL Summaries 


S& PT4 — Experimental Design 


* Goal: Evaluate candidate Detect and Avoid (DAA) displays and algorithms with 
respect to traffic avoidance and collision avoidance. 
— What are the appropriate alerting thresholds for self separation? 
— What are the minimum information requirements for DAA displays? 
— Is there a performance difference between integrated and standalone displays? 
— What advanced display features improve pilot performance on maintaining well 
clear from other traffic? 
¢ What advanced display features improve pilot performance on maintaining 
well clear from other traffic? 
— Experimental Design: Mixed Factorial Design 
— 2 (Display: Standalone, Integrated) 


X 2 (Information: Basic, Advanced) 
— X 2 (Self-Separation Alerting Threshold) 


S& PT4 — Display Configurations 


Display Location Level: Standalone versus Integrated 
— Standalone: display located on own monitor, separate from primary moving map 
display 
— Integrated: display integrated with primary moving map display 
Display Information Level: Basic versus Advanced 
— Basic presents minimum information requirements only 


— Advanced information elements: 
¢ Implementation different between Standalone and Integrated displays 
¢ Additional informational elements 
— Additional alerting level (predictive CA) 
— Time to and location of predicted CPA (intruder and ownship) 
— Vertical situation display (Integrated only) 
¢ Maneuver guidance 
— Suggestive: Trial/vector planner 
— Directive: Maneuver recommendations 
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PT4 — Standalone Displays 


Advanced 


PT4 — Integrated Displays 
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Basic . Advanced 
Information 


There was a significant main effect of 
Information on Total Response Time, p < .05 
— Advanced was significantly faster (by 13.79 
seconds on average) compared to Basic 
Pilots took an average of 37.87 seconds to 

complete their final edit in response to 
DAA/CA alerts (from first alert appearance) 
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PT4 — Total Response Time Results 


37.14 38.60 


Standalone . Integrated 
Display 


There was no significant main effect of 
Display on Total Response Time, p > .05 
Pilots took an average of 37.87 seconds to 
complete their final edit in response to 
DAA/CA alerts (from first alert appearance) 


S& PT4 — Losses of Well Clear 
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e There was not a significant main effect of Information on Proportion of Losses of Well 
Clear, p > .05 


° On average pilots failed avoid a loss of well clear 44% of the time 
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S& PT4 — Results Summary 


* Consistent advantage seen for Advanced over Basic displays in pilot response 
times 

¢ No significant differences in proportion or severity of losses of well clear, 
however, advanced trended toward lower rates of LoWC than basic 

e There were no significant differences between the Standalone and Integrated 
condition 
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S& iHITL — Experimental Design 


¢ Goal: Determine the individual contributions of the various PT4 advanced 
display features to pilots’ response times and ability to maintain well clear 
¢ One-Way Repeated Measures Factorial: Display Information Level (4 Level; 
Within Subjects) 
— D1: Advanced Display with Information Only (Informative) 
— D2: Advanced Display with Information + Vector Planner (Suggestive) 
— D3: Advanced Display with Information + Auto Resolutions (Directive) 


— D4: Advanced Display with Information + Vector Planner + Auto Resolutions 
(Suggestive + Directive) 
¢ Roughly same as ‘Advanced’ suite in PT4 
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iHITL — Display Conditions 


S&S iHITL — Total Response Time Results 


Mean Time (s) 


Info Only Info + Vector Info + AR Info + Vector + AR 
Display 


There was not a significant main effect of Information on Total Response Time, p > .05 


Pilots took an average of 14.92 seconds to complete their final edit in response to 
DAA/CA alerts (from first alert appearance) 
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S& iHITL — Losses of Well Clear 


0.227 


Proportion 


0.053 


Info Only Info + Vector Info + AR Info + Vector + AR 
Display 


¢ There was nota significant main effect of Display Configuration on Proportion of Losses 
of Well Clear, p > .05 


¢ On average pilots failed avoid a loss of well clear 13% of the time 
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S& iHITL — Results Summary 


Total Response Time: 
— No significant differences between displays 


— Trend shows Info + AR and Info + Vector + AR as faster than Info Only and Info + 
Vector 


Well Clear Metrics: 
— No significant differences between displays 
— Info Only and Info + Vector display conditions had 2.5X as many LoWCs than the 
Info + Vector + AR 
Overall: 


— The two displays with directive guidance (i.e., Auto-Resolutions) performed better 
than the informative only and suggestive only displays 

— This result is confounded by integration of Auto-Resolutions tool with auto-pilot 
display 
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S& PTS — Overview 


* Goal: Continue evaluation of candidate Detect and Avoid (DAA) displays and 
algorithms with respect to traffic avoidance and collision avoidance to inform 
SC-228 DAA Minimum Operational Performance Standards 


¢ Method: 

— Build upon results of previous hitl simulations results and lessons learned to 
identify minimum DAA display and guidance requirements for draft SC228 MOPS 

— Take into account SC-228 decision that directive guidance would not be part of the 
minimum requirements by focusing on various suggestive guidance displays versus 
informative 

— Address issues of integrating DAA display features into the GCS control and 
navigation functionality 


S& PTS — Experimental Design 


e Mixed Factorial Design 

— Display Configuration (Within-Subjects Independent Variable): 
* Configuration 1: Minimum Information Set (No Guidance) 
° Configuration 2: Stratway+ No Fly Bands 
¢ Configuration 3: JADEM Omni Bands 
¢ Configuration 4: JADEM Vector Planning Tools 

— Sensor Performance (Between-Subjects Independent Variable) 
¢ Level 1: Perfect Surveillance Data 
¢ Level 2: Imperfect Surveillance Data 
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PT5 — Display Conditions 


Threat Type at First Alert 


Mean 


24.04 


© Median 


Warning 


Corrective 


Info Only 


PT5 — Total Response Time Results 


Threat Type at First Alert 


® Corrective Mean 


@ Warning Mean 


Omni Bands Vector Planner 


Stratway+ 
Configuration 


e Pilots responded, on average, 10s faster to SS Warning Alerts than they did to 


Corrective SS Alerts 


— Pilots exhibited less variability between displays when responding to DAA Warning Alerts 


than to Corrective DAA Alerts 


* Range for DAA Warning Alerts: 11s - 15s 


* Range for Corrective DAA Alerts: 19s — 30s 
— Variability due to coordination with ATC — adds ~ 10 secs to total response time 
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S& PT5 —Losses of Well Clear 
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There was a significant main effect of Display Configuration on Proportion of Losses of 
Well Clear, p < .05 


— Pilots in the Omni Bands condition had significantly fewer losses of well clear than those in 
the Info Only condition 


On average pilots failed avoid a loss of well clear 4% of the time 
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S& PTS — Results Summary 


° Suggestive guidance in the form of banding resulted in safer and more timely 
maneuvers away from conflicts 
— Lower overall proportion of LoWC for both banding displays (none for omni bands) 


— Least severe LoOWC for both banding displays; most severe with info only (where an 
NMAC occurred) 


— Shorter Total RTs for both banding displays 
— Pilots self-report as preferring the banding displays 
¢ Results support decision for suggestive guidance as a minimum information 
requirement for DAA displays 


— Although Vector Planner display had performance between info only and banding 
displays, design approach not consistent with good HF principles and very poor 
performance compared to Omni Bands (despite same underlying algorithm) 


PTS — Results Summary 


Soe 


¢ PT5 results also inform the DAA alerting structure and thresholds: 


Pilots responded consistently to a DAA Warning alert (no ATC coordination 
required) in ~ 15 seconds 

Responded to a DAA Corrective alert (ATC coordination is required) in ~ 25 
seconds, though more variability 

Therefore, ATC coordination adds approximately 10 seconds to DAA timeline 


33 


Implications of first three HITLs 
on Draft DAA MOPS 


S& Overview Summary of HITLs 
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Total Response Times Across Simulations 


PT4 
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Losses of Well Clear Proportions 
Across Simulations 
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S& Draft MOPS Informed by HITLs 


e Maneuver guidance in the form of bands 


¢ Alerting structure with DAA Warning and Corrective alerts and their 
corresponding pilot actions 


¢ Minimum thresholds for alerting levels 
¢ Minimum surveillance range for onboard radar 
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Draft MOPS Informed by HITLs: 
Suggestive Maneuver Guidance 
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Draft MOPS Informed by HITLs: 
Alerting Levels and Thresholds 


F . Alerting Time Aural Alert 


Immediate action required 25 sec 
DAA Warning Alert Notify ATC as soon as practicable after taking (TCPA approximate: 
action 60 sec) 


“Traffic, Maneuver 
Now” 


‘ ; . 55 sec 
On current course, corrective action required PrePAsppremimats: 
Coordinate with ATC to determine an PP : 


Alert : 90 sec) 
appropriate maneuver 


a On current course, corrective action should 


DAA Corrective “Traffic, Avoid” 


55 sec 
(TCPA approximate: “Traffic, Monitor” 
90 sec) 


DAA Preventive not be required 
Alert Monitor for intruder course changes 
Talk with ATC if desired 


No action required 
Guidance Traffic Traffic generating guidance bands outside of 
current course 


[4] Remaining Traffic No action expected 


Draft MOPS Informed by HITLs: 


Surveillance Range 


~90 sec 


~80 sec 


Time until CPA 


Latency TC Interaction Pilot Aircraft 
Time Response Time Maneuver Time 
(~10 sec) (~15 sec) (~30 sec) 
TOTAL RESPONSE T|ME: 
A . . : 
pproximate detection ‘tude ‘Approximate detection 


range = : . 
ange = 8 Blots Determine Resolution 
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Negotiate Clearance with ATC and uplink 
maneuver to aircraft 


Well Clear 
Threshold 
(~35 sec) 


NMAC 
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V&V HITL (PT6) 


S& PT6 — Overview 


Purpose: 
— Conduct final V&V activity in support of SC-228 DAA HMI requirements for 
displays, alerting and guidance 
— Determine if pilot performance w/ minimum requirements (as currently defined in 
the draft MOPS) comparable to previous simulations, such as Part Task 5? 
Goals: 
— Implement the minimum display, alerting and guidance requirements as close as 
possible in simulation 
— Test in representative simulated flight environment 
¢ E.g., airspace w/ ATC in-the-loop, multiple UAS missions, secondary tasks, high-fidelity 
surveillance models, TCAS II 
— Expected outcome/product(s): pilot performance data to validate final DAA MOPS 
¢ Losses of Well Clear 
¢ Pilot response times 
¢ Additional pilot behavior: TCAS compliance, type/size of maneuvers, ATC coordination 
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S& PT6 — Experimental Design 


Mixed Factorial 
— Display Configuration (within-subjects) 
¢ Standalone DAA display (decoupled from moving map/TSD) 
¢ Integrated DAA display (collocated with moving map) 
— Ownship Equipage (between-subjects) 
¢ TCAS Il-equipped 
* No TCAS Il 
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PT6 — Experimental Design 


ea sansa Pilot Action Buffered Well Time to Loss of Well Aural Alert 
Y Clear Criteria Clear Verbiage 
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DAA Warning 
Alert 


Corrective DAA 
Alert 


Preventive DAA 
Alert 


Guidance Traffic 


Remaining Traffic 


Immediate action required 

Comply with RA sense and vertical rate 
Notify ATC as soon as practicable after 
taking action 


Immediate action required 
Notify ATC as soon as practicable after 
taking action 


On current course, corrective action 
required 

Coordinate with ATC to determine an 
appropriate maneuver 


On current course, corrective action 
should not be required 
Monitor for intruder course changes 
Talk with ATC if desired 


No action required 
Traffic generating guidance bands 
outside of current course 


No action required 
No coordination required 


*DMOD = 0.55 nmi 
*ZTHR = 600 ft 
*modTau = 25 sec 
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HMD = 0.75 nmi 
ZTHR = 450 ft 
modTau = 35 sec 
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60 sec) 


55 sec 


(TCPA approximate: 


90 sec) 


55 sec 


(TCPA approximate: 


90 sec) 


“Climb/Descend” 


“Traffic, 
Maneuver Now” 
x2 


“Traffic, Avoid” 


“Traffic, Monitor” 


f aah PT6 — Display Conditions 


Standalone Configuration 


| See. 


DAA Display | TSD Side Panel 


Notes: 
e Pilot could only make uploads via TSD; DAA Display only served as a traffic 
reference 


¢ Pilots trained on how to adjust orientation on both DAA & TSD displays 
¢ North Up vs. Track Up, and whether orientations matched, was up to pilot discretion 
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PT6 — Display Conditions 


Integrated Configuration 


TSD w/ DAA Display 


Side Panel 


S& PT6 — TCAS II Effects 


Proportions of LoWC 


TCAS II equipage was split between first half and second half of data collection 
— No significant main effect on proportion of LoWC 
— No significant main effect on total response time 


Remaining results are collapsed across the TCAS variable 


Proportion of LOWC by TCAS Total Response Time by TCAS 
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S& PT6 — Response Time Results 


PT6 to PT5 comparison uses aircraft response time instead of total response 
time 
— Difference is whether we are measuring at the first vs final upload 
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PT6 — Aircraft Response Time 


e Pilots sent their first upload to their aircraft 1 sec faster in the Integrated display 
configuration (statistically significant; p < .05) 
— More pronounced difference between displays when separated by alert level 


¢ Pilots sent final upload 5.5 sec faster (~30%) in response to DAA Warning alerts in Integrated display 
configuration 


¢ Comparable to PT6 results, although response to Warning in Standalone configuration 
is Slower in PT6 
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PT5 —Losses of Well Clear 
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e There was not a significant main effect of Display Configuration on Proportion of Losses 
of Well Clear, p < .05 


¢ Proportion of LOWC where pilot was at fault (i.e., had enough time to respond) lower 
in PT6 than all conditions in PT5 except for Omni Band 
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S& PTS — Results Summary 


¢ Overall pilot performance was consistent with previous simulations 
when using minimum display, alerting & guidance requirements 
— Onsome measured response metrics performance was slightly better 
— Proportions of LoWC virtually identical 
— Standalone display resulted in little to no performance differences compared to 
the Integrated display configuration 


¢* Data supported display, alerting & guidance requirements as 
previously drafted 


Summary of Contributions to Phase 1 MOPS 


S& Summary of NASA HITL Contributions to SC-228 


¢ Suggestive DAA guidance requirements 
¢ Alerting logic and thresholds 
e Integrated or standalone* 
Oe PT4, iHITL, PTS, PT6 HITLs 
e Pilot response timeline 
— Derived RADAR Requirements 


¢ V&V of alerting, guidance and display draft MOPS 


TCAS/DAA interoperability concept 
— Requirements for DAA guidance and alerting 


DAA-TCAS 
Interoperability HITL 
Well Clear Recovery guidance logic/display 


Alerting and guidance logic for special cases 


Special Cases 
— E.g., no altitude, no bearing Mini HITL 


Alerting and guidance displays for special cases 


Questions? 


